{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/uptrain-ai/uptrain/blob/main/examples/managed_service_client_tutorial.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 align=\"center\">\n",
    "  <a href=\"https://uptrain.ai\">\n",
    "    <img width=\"300\" src=\"https://user-images.githubusercontent.com/108270398/214240695-4f958b76-c993-4ddd-8de6-8668f4d0da84.png\" alt=\"uptrain\">\n",
    "  </a>\n",
    "</h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using UpTrain with our Managed Service API Client\n",
    "\n",
    "You can use UpTrain's managed service to log and evaluate your LLM responses. Just provide your UpTrain API key (no need for OpenAI keys) and UpTrain manages running evaluations for you with real-time dashboards and deep insights."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install UpTrain\n",
    "\n",
    "Run the following commands in your terminal to install UpTrain:\n",
    "```bash\n",
    "pip install uptrain\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from uptrain import APIClient, Evals, CritiqueTone\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create an UpTrain API Client\n",
    "\n",
    "Before we can start using UpTrain, we need to create an API client. You can do this by passing your API key to the `APIClient` constructor.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "UPTRAIN_API_KEY = \"up-***************\"\n",
    "client = APIClient(uptrain_api_key=UPTRAIN_API_KEY)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create your data\n",
    "\n",
    "You can define your data as a simple dictionary with the following keys:\n",
    "\n",
    "- `question`: The question you want to ask\n",
    "- `context`: The context relevant to the question\n",
    "- `response`: The response to the question"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [{\n",
    "    'question': 'Which is the most popular global sport?',\n",
    "    'context': \"The popularity of sports can be measured in various ways, including TV viewership, social media presence, number of participants, and economic impact. Football is undoubtedly the world's most popular sport with major events like the FIFA World Cup and sports personalities like Ronaldo and Messi, drawing a followership of more than 4 billion people. Cricket is particularly popular in countries like India, Pakistan, Australia, and England. The ICC Cricket World Cup and Indian Premier League (IPL) have substantial viewership. The NBA has made basketball popular worldwide, especially in countries like the USA, Canada, China, and the Philippines. Major tennis tournaments like Wimbledon, the US Open, French Open, and Australian Open have large global audiences. Players like Roger Federer, Serena Williams, and Rafael Nadal have boosted the sport's popularity. Field Hockey is very popular in countries like India, Netherlands, and Australia. It has a considerable following in many parts of the world.\",\n",
    "    'response': 'Football is the most popular sport with around 4 billion followers worldwide'\n",
    "}]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Log and evaluate your data\n",
    "\n",
    "Now that we have our data, we can log it and evaluate it using UpTrain. We use the `log_and_evaluate` method to do this. This method takes the following arguments:\n",
    "\n",
    "- `project_name`: The name of your project\n",
    "- `data`: The data you want to log and evaluate\n",
    "- `evals`: The evaluations you want to perform on your data\n",
    "\n",
    "You can find the list of all available evaluations [here](https://docs.uptrain.ai/key-components/evals)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m2023-09-29 19:25:43.139\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36muptrain.framework.remote\u001b[0m:\u001b[36mlog_and_evaluate\u001b[0m:\u001b[36m446\u001b[0m - \u001b[1mSending evaluation request for rows 0 to <50 to the Uptrain server\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "results = client.log_and_evaluate(\n",
    "    project_name=\"Sample-Project\",\n",
    "    data=data,\n",
    "    checks=[Evals.CONTEXT_RELEVANCE, Evals.FACTUAL_ACCURACY, Evals.RESPONSE_RELEVANCE, CritiqueTone(persona=\"teacher\")]\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get your results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "   {\n",
      "      \"question\": \"Which is the most popular global sport?\",\n",
      "      \"context\": \"The popularity of sports can be measured in various ways, including TV viewership, social media presence, number of participants, and economic impact. Football is undoubtedly the world's most popular sport with major events like the FIFA World Cup and sports personalities like Ronaldo and Messi, drawing a followership of more than 4 billion people. Cricket is particularly popular in countries like India, Pakistan, Australia, and England. The ICC Cricket World Cup and Indian Premier League (IPL) have substantial viewership. The NBA has made basketball popular worldwide, especially in countries like the USA, Canada, China, and the Philippines. Major tennis tournaments like Wimbledon, the US Open, French Open, and Australian Open have large global audiences. Players like Roger Federer, Serena Williams, and Rafael Nadal have boosted the sport's popularity. Field Hockey is very popular in countries like India, Netherlands, and Australia. It has a considerable following in many parts of the world.\",\n",
      "      \"response\": \"Football is the most popular sport with around 4 billion followers worldwide\",\n",
      "      \"score_context_relevance\": 1.0,\n",
      "      \"explanation_context_relevance\": \"1. The question asks for the most popular global sport.\\n2. The extracted context provides information about the popularity of various sports, including football, cricket, basketball, tennis, and field hockey.\\n3. The extracted context states that football is undoubtedly the world's most popular sport, with major events like the FIFA World Cup and sports personalities like Ronaldo and Messi drawing a followership of more than 4 billion people.\\n4. Based on the information provided in the extracted context, it can be concluded that football is the most popular global sport.\\n5. Therefore, the extracted context can answer the given question completely.\\n\\nScore: 1.0\\n1.0\",\n",
      "      \"score_factual_accuracy\": 1.0,\n",
      "      \"explanation_factual_accuracy\": \"1. Football is the most popular sport.\\nArgument for yes: The context explicitly states that football is undoubtedly the world's most popular sport.\\nArgument for no: The context mentions other sports like cricket, basketball, and field hockey that are also popular, so it is not clear if football is the most popular.\\nJudgement: yes. The argument for yes is stronger as the context clearly states that football is the most popular sport.\\n2. It has around 4 billion followers worldwide.\\nArgument for yes: The context mentions that football has a followership of more than 4 billion people.\\nArgument for no: No arguments.\\nJudgement: yes. The argument for yes is stronger as the context explicitly states that football has more than 4 billion followers worldwide.\\n\",\n",
      "      \"score_response_relevance\": 0.0,\n",
      "      \"explanation_response_relevance\": \"1. Read the question: \\\"Which is the most popular global sport?\\\"\\n2. Read the response: \\\"Football is the most popular sport with around 4 billion followers worldwide.\\\"\\n3. Compare the response to the question.\\n4. The response directly answers the question by stating that football is the most popular global sport.\\n5. There is no additional irrelevant information in the response.\\n6. The response is concise and does not contain any unnecessary details.\\n7. The response is focused and does not include any logical fallacies, incorrect assumptions, or errors in reasoning.\\n8. Based on the above analysis, the generated answer has no additional irrelevant information.\\n\\nThe correct answer is C. The generated answer has no additional irrelevant information.\\n0.0\\n0.0\",\n",
      "      \"score_tone\": 0.4,\n",
      "      \"explanation_tone\": \"The tone is not appropriate for a teacher. It is a simple statement of fact without any context or explanation. A teacher would typically provide more information and engage the student in a discussion about the topic.\"\n",
      "   }\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(results, indent=3))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
